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The literature on program evaluation contains numerous 
suggestions that evaluative information is frequently 
underutilized or inappropriately utilized by administrators. 
This paper reviews the literature on utilization with a view 
toward identifying workable strategies for optimizing 
appropriate use. An effort is made to synthesize previous 
thought and empirical research .n utilization. Weaknesses in 
previous naturalistic and simulat.ion use studies are discussed. 
A research agenda for the future is presented. 
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A sense of frustration permeates the literature (King, 
T^^tnpson & Pechman, 1981) on the use of program evaluation 
^'^^'^rmation . As Weiss ( 1972, p. 318) notes, evaluation's 

primary justification is that it contributes to the 
rationalization of decision-making. Although it can 
serve such other functions as knowledge-building and 
theory-testing, unless it gains serious hearing when 
program decisions are made, it fails in its major 
purpose. 

Howey^j^^ it is clear that "there is something basically wrong 
"^^^ evaluation" (Orlandi & Conslave, 1977, 3). House 

^^^''^^j p. 4) argues that "even under favorable crcumstances 
^^^■''Uation data might account for only 20% of a decision." 

Reading the literature on use of evaluative information 

c 3 n 

contribute to chronic depression. For example, Haenn 
^^^^0, p. 2) concludes that "evaluation results generally have 
«d neither as a means of judging program results nor as a 
S^^^e to program improvement." Williams and Evans (1969, 
conclude that, "in the final analysis, the test of the 
^^^^otiveness of outcome data is its impact on implemented 
polidy^ By this standard, there is a dearth of successful 
®^3lv4ation studies." Alkin and Daiiiak (1979, p- 41) conclude 
that nthere have been great hopes for evaluation, not only 
among evaluators themselves, but also among other educators, 
elected officials, and the public. Yet these hopes have 
dimm^cl." Worthen and Sanders (I973, P. 1) conclude that 
"^^^luation is one of the most wi.. oiy discussed but little used 
P^'o^^sses in today's educational systems." Wholey, Scanlon, 
P^ffy, Fukumoto, and Vogt (1970, P- ^6) conclude that "the 
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recent literature is unanimous in announcing the general 
failure of evaluation to affect aecision-making in a 
significant way." Rippey (1973, p. 9) concludes that "at the 
moment, there seems to be no evidence that evaluation, although 
the law of the land, contributes anything to educational 
practice other than headaches for the researcher, threats for 
the innovators, and depressing articles for journals devoted to 
evaluation." Finally, Stake (1973, P^ 314) concludes that "we 
do not know whether or not evaluation is going to contribute 
more to the problems of education or more to the solutions." 



This characterization apparently applies equally well to 
judicial (Saks, ''^^ and legislative settings (Brandl, 1980; 
Mitchell, 1980). Unfortunately, several directors of 
evaluation from local education agencies (LEA's) have suggested 
that the characterization may generalize to those settings too: 



In an ideal world we wouldn't have to worry about 
utilization. Educators would be eagerly awaiting our 
findings and would promptly rush to put them into 
practice. I don't need to tell you- "that isn't 
happening (Holley, 1979, p. 2). 

The apparent nonuse of evaluation findings is one of 
the most vexing problems associated with the practice 
of evaluation in the public school setting. Even in 
districts with active evaluation sections it is 
difficult to establish a direct correspondence 
between evaluation results and educational decisions 
(Novak, 1977, p. 1). 

All LEAs, with possibly a few exceptions, can point 
to their volumes jLf. research and evaluation verbiage 
setting on the shelves of district administrators 
being used for little else than a door stop, swatting 
flies, or any of the other various and sundry 
purposes for which research is used in the public 
schools (Kilbourne & DeGracie, 1979, P- '2). 
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The Tragedy of Non-Use 

The non-use of evaluative information, when that use would 
be appropriate, is tragic. Non-use represents an enormous 
waste of effort. As Datta (1979, p. 22) notes, "considerable 
effort is involved in conducting almost any evaluation: in 
identifying the evaluation question, in designing the study, in 
overcoming the obstacl^s^ to conducting ao evaluation and 
protecting it methodologically from uninterpretability 

Non-use also represents waste of monies. For example, in 
197^ direct expenditures on non-defense evaluation projects by 
the federal government alone amounted to $146 million (Kelezo, 
1974). Surprisingly, "more and more money is being invested in 
evaluation studies at the same time that we are questioning 
their results and effects" (Raizen, 1978, p. 3). The indirect 
costs of non-use, when ineffective programs are not modified or 
discontinued, are even more staggering: 

The utilization of research crisis concerns the 
spending of billions of dollars in private and public 
funds to fight problems of poverty, disease , 
joblessness , mental anguish , cr ime , hunger and 
inequality (Patton, 1978, p. 12). 

However, the greatest tragedy is that non-use means that 
the clients of educational and social programs receive less 
than optimally effective help. Failure to utilize evaluative 
information is tragic because, as Wise (1980, p. 16) notes, "no 
one else is given the resources and time to question, observe, 
assess, weigh, probe, and reflect that the evaluator is given." 
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Purposes of the 

Weiss (1979, i) summarizes the literature on evaluation 
use in this aianner. 

Studying ^he effects of social science research and 
evaluation used to be a problematic enterprise, and 
anyone Writing on the subject had to Justify its 
importance... We can [now] take the significance of 
the questions for granted. The challenge is to get 
on with the job. 

This papej.»s P^'^Poses are to synthesize previous theoretical 
and ernpij-ical bindings regarding evaluation use and to propose 
an agenda for ^^utu^e research on evaluation use. 

£H|at: the rationality of non-u se 

It must be aci^j^o^^iedged , however, that the non-use of 
evaluative in^oi"niation can be quite rational, for a number of 
reasons, for example, it is clear that some evaluation studies 
are Poot^ done gnd do not merit use. Ironically, there is 
empirical evidence (Alkin, Kosecoff, Fitz-Gi'jbon & Seligman, 
197^» P, il8) that poorer quality evaluations may be performed 
on projects which quality evaluations could most help to 
improve. At any ^ate, as Cuba and Stufflebeam (1970, p. 6) 



observe : 



Many '"^Searchers make wrong assumptions about what an 
gvaiuatiorj study should accomplish, and — [then] 
based on thes -roneous assumptions, ''^searchers 
foist jjgjj au.ice upon unsuspecting and 
unsophisticated practitioners. As a consequence, 
evaiuatiojjs are usually useless, and practitioners 
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are largely justified in the jaundiced view they 
typically have taken about evaluation and its 
utility. 

A common situation which justifies non-use occurs when 
evaluato""S do not attend to a program's actual degree of 
implementation. As Guttentag and Struering (1975, p. 4) 
observe, "obvious though it may seem, evaluations continue 
without either raising or answering the primary question: 
'Does the program [even] exist?"' Unfortunately, as Williams 
and Elmore (1976, p. xii) note, "ignoring implementation has 
been equally disastrous for research and analysis." This 
situation continues despite the availability of several 
stratgies for measuring implementation prior to comparison of 
results for program participants and non-participants (Revicki 
& Rubin, 1980), including most notably the model developed by 
Hall and Loucks (1977). 

Practitioners also often feel that evaluation studies do 
not merit use because the evaluation results contradict the 
practitioner's intuitions regarding program impacts. As Cuba 
(1969, p. 1) notes, "for decades the evidence produced by the 
application of conventional evaluation procedures has 
contradicted the experential evidence of the practitioner. 
Innovations have persisted in education not because of the 
supporting eviden-a of evaluation but despite it." This 
frequently occurs when summative evaluations find "no 
significant differences" associated with a program — certainly a 
common result- -lu;; the practitioner's experience suggests, that 



ERIC 



Page 8 



the program really did substantially alter classroom life* As 
Shapiro (1973, p. 527) argues, "while it is important to try to 
explain negative [i.e., statistically non-significant 
summative] test results, it is far more important to account 
for the disparity between the negative test findings and the 
clear differences observed in classroom behavior." 

There is also evidence that non-use is rational, from the 
practitioner's perspective, since the practitioner may not view 
the world from within a scientific paradigm, and differences in 
the evaluator * s and the practitioner ' s perspectives may reduce 
trust and impede effective communication . As Deal and Rallis 
(1980, p. 216) explain: 

Theoretically, collaborative relationships require a 
shared perspective, high trust and power parity. The 
existing relationship between knowledge producer and 
user, however, is often characterized by different 
perspectives, low trust and an asymmetrical 
distribution of power. 

Empirical research makes clear how this situation might occur 
in public school settings: 

Not only have only 42% of them [LEA evaluation unit 
heads] not taught, but 70% have not run a school. 
This means that even when evaluation heads have 
teaching backgrounds, they do not take the typical 
advancement route to the central office (Lyon , 
Doscher, McGranahan & V/illiams, 1978, p. 66). 

There is also evidence that non-use may be rational when 
administrators invoke latent institutional or personal goals , 
which are quite distinct from formal program goals , as 
standards for determining program merit > For example, 
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Granville (1977, p. 2) explains that "a decision maker, in 
addition to considering whether or not a program has fulfilled 
its manifest objectives, must also consider its fulfillment of 
latent objectives, such as enhancing the agency's prestige or 
expanding its resources." It is also important to remember that 
administrators may have their own survival needs which can 
affect their use of evaluative information: 



High level administrative tenure is quite short as is 
that of elected officials. Policy makers must 
demonstrably show actions [sic] in a short period of 
time as constituencies are not willing to wait 
(Mathis, 1980, p. 2). 



Finally, it is important albeit painful to acknowledge 
that some evaluation studies were never meant to be used. As 
Alkin (1976, p. 16) explains, this is frequently the case when 
an externally, mandated evaluation is involved: 



Many practicing school administrators... believe 
that evaluation is simply an event that leads to 
compliance with various agency requirements. There 
is no real expectation that major basic decisions 
will be made. The name of the ballgame is simply not 
to get "dinged" by the governmental agency. 

Alkin (1980a, p. 3) more recently made the same point by way of 

analogy: 



And, to pursue the analogy, suppose the host at this 
garden party should insist that each of the guests 
periodically rate the quality of the party, or the 
drinks, or the food etc. — it can't really be expected 
to have much impact. This somewhat peculiar, 
externally imposed requirement will be tolerated as 
part of the "price of admission," so to speak, but it 
won't really change the behavior of individuals. 

Indeed, deliberate non-use is thoroughly rational if programs 
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are not conceptualized well enough to possess "evaluability" 
( Rutrnan , 1977) • 

Taken together, these four caveats suggest two important 
conclusions. First, we must not expect too much of evaluation, 
at least as it is currently practiced (Daillak, Alkin & White, 
1978, p- 10). McClean (1979, p. 26, emphasis in original) 
helps to put this matter into perspective: 

Schools are overdetermined ; that is, they are shaped 
by iiany forces, more even than are necessary to make 
them the way they are. Take away or change one force 
and nothing in a school may change. 

Stevens and Tornatzky (1980, p. i^O) concur, although for a 

different reason. They argue that underutilization should be 

expected, aince "underutilization of Knowledge is actually 

quite common with an innovation such as program evaluation." 

Second, and more important, however, it must be remembered 

that 

Evaluation has been ignored, misused, and overused, 
as well as appropriately used in policy and decision 
making. Our task is to make more appropriate uses of 
evaluative information (Braskarap 4 Brown , 1980, 
p . x) . 

Thus Caplan (1980, p. 5) argues that "there is a real danger in 
uncritically accepting utilization as desirable or in being 
oversold on its value. Not all utilization is good and not all 
nonutilization is bad." Similarly, Davis and Salasin (1975, 
p. 622, emphasis in original) argue that "it is a clear lesson 
that concern over utilization must emphasize appropriate use 
rather than just greater use." 
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THE CONCEPT OF USE 
Underestimation of Use Levels 

Recently several authors have suggested that the extent of 
evaluation use may be underestimated in the literature 
(cf. Datta, 1978, p. 3). For example, Wise (1978, p. 2H) has 
suggested that "if there is an evaluation utilization problem, 
it is not that decision-makers do not use the information they 
receive, it is that evaluators cannot easily see their 
information being used in the incrementalism of real-world 
decision-making." Daillak, Alkin, and White (1978, p. 1) 
concur, noting that "the few empirical studies that have been 
conducted, however, seem to present a picture which is less 
gloomy than many of the more speculative articles; they 
suggest that evaluation can have impact upon decision-making, 
although not necessarily the kind of dramatic go/no go 
influence some would wish." 

There is empirical evidence that administrators do value 
evaluative information, although they might want changes in the 
methodology of some evaluations. For example, Alkin, Kosecoff, 
Fitz-Gibbon, and Seligman (1974, p. 28, emphasis in original) 
report that "project directors found evaluations most useful in 

identifying possible problem areas No project director 

indicated that he would prefer not to have an evaluator on the 
project." However, another important empirical study suggests 
that results of use studies gener-a^ly must be interpreted with 
some caution: 

The results of this study seemed at first to raise 
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some doubt about the fundamental assumption 
underlying the study: the assumption that evaluation 
is an exercise in the service of Jecision-maker s • 
While all those interviewed were, without exception, 
decision-makers in one capacity or another, their 
requests for reports were not always connected with 
the decisions they had to make. Frequently, they 
wanted the reports so as to inform the people above 
them or below them in the administrative heirarchy of 
theii" government agency (Brickell, Aslanian & Spak, 
197^, p. 56). 



In any case, it is clear that the quality of utilization 
research and of estimates of use levels presumes an acceptable 
definition of use. As Tittle (1977, p. 3) notes, "papers 
concerned with the analysis of impact and impact assessment 
methodology have not always dealt with the problem of how to 
define impact." Patton (1978, p. 32) goes so far as to argue 
that "the predominant image of nonutil ization that 
characterizes much of the commentary on evaluation research can 
be attributed in substantial degree to a definition of 
utilization that is too narrow in its emphasis on seeing 
immediate, direct, and concrete impact on program decisions." 
Thus Alkin (i980a, p. 5) concludes that: 



From among these continuing strains of 
non-utilization, there now seems to be emerging a new 
methodolgy which points to instances in which 
evaluation information is in fact used. Moreover, 
these new results are not really contradictory with 
Prior data on utilization ; the [new] evidence on 
utilization rests upon a broader definition of 
utilization and different categories of evaluative 
information." 

Similarly, Alkin, Daillak, and White (1979, p. 16, emphasis in 
original) argue that 



taken together , the studies and our observations and 
experiences suggest to us that evaluation can make a 
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difference, that it does so more often than the 
published critiques suggest, that some school 
districts characteristically produce a high 
proportion of useful evaluations, and that some 
evaluators have acquired skills that allow them to 
carry out technically competent and programmatically 
influential evaluations. 

In a fine piece on utilization, Weiss (1979, p. 13, 

emphasis removed) argued that "until we resolve questions about 

the definition of use, we face a future of non-comparable 

studies of use and scant hope of cumulative understanding of 

how evaluation and decision-making intersect." Most of the 

agitation for an improved conceptualization of use stems from a 

growing recognition that use generally does not take very 

direct or dramatic forms. as Alkin (1979, p. 3) suggests, "it 

is not enough to ask in September what the effects of the 

previous academic yearns evaluation have been; as our 

illustration suggests, it may take two or three or more years 

before major program cnanges occur." Thus Andrews (1979, p. 18) 

concluded that, "Of great importance is the finding that 

evaluations tend to have small incremental impacts on the 

programs; the 'big bang' theory of evaluation impact should 

apparently be discarded." 

Types of Use 

The literature reflects a growing recognition that subtle 
but still important types of use may be more typical than are 
direct, radical program changes (Brown & Braskamp, 1980, 
p. 92). As '.-^eiss (1977, p. 534) suggests, "government 
officials use research less to arrive at solutions than to 
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orient themselves to problems... And [even] much of this use 
is not deliberate, direct, targeted, but a result of long term 
percolations of social science concepts, theories and findings 
into the climate of informed opinion." 

Several conceptualizations of types of use have been 
offered. for example, Fullan CT979) has suggested that 
information may result in changes in values, in understanding, 
in roles, in organization, or. in materials. Some authors use 
different terms for types of use which apparently involve the 
same processes. These terms and the concepts they represent 
are presented in Table 1. As Weiner, Rubin, and Sachse (1977, 
p. 12) observe, "these categories are neither mutually 
exclusive nor exhaustive." 

"Instrumental" use represents the more traditional view of 
use of evaluative information, e.g., the information results in 
go/no go decisions regarding program termination. This type of 
use rarely occurs, although some examples of "instrumental" use 
can certainly be identified (cf. Alkin, Daillak & White, 1979, 
p. 22U). "Conceptual" use is in practice a more common form of 
use. An example of conceptually targeted evaluation is 
provided by Cook (1974). Cook argued that "Sesame Street" 
should not be evaluated merely on the basis of the program's 
impacts on the reading readinecs of children. It was suggested 
that the program's impacts on the gaps in reading readiness 
between lower socio-economic status children and other children 
should also be considered; this argument changed the conceptual 



Table 1 
Types of Use 



Term 



Synonomous Labels 



Definition 



"Instrunental" (Rich, 1977) 



'Conceptual" (Rich, 1977) 



'Spbolic" (Weiner et al., 
1977, p. 13) 



Ritualistic" (Braskamp, 
1980, p. 1) 



"Allocative" (Weiner et al., 
1977, p. 12) 



"Appreciative" (Weiner et 
al., 1977, p. 14) 

"Enlightenment" (Braskamp, 
1980, p. 1) 



"Persuasive" 



"Anticipatory" 



"results lead to a decision 
about program allocations, 
expansion, and elimination" 
(Braskamp, 1980, p. 1) 

"influencing a policpaker's 
thinking about an issue 
without putting information 
to any specific, docuraentable 
use" (Rich, 1977, p. 200) 

"drawing on evaluation evidence 
in attempts to convince others 
to support a political position, 
or to defend such a position fi-om 
attack" (Leviton & Hughes, 1979, 
p. 7) 

"the impact that evaluation has 
upon the diligence exhibited by 
program participants" (Weiner et 
al., 1977, p. 12) 
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frame of reference for evaluating the program. Clearly, 
"conceptual" use of evaluation can have dramatic impacts at 
times. It is also clear that sometimes "it is difficult to 
determine where conceptual use ends and instrumental use 
begins" (Leviton & Hughes, 1979, p. 10). 

"Symbolic" use is common and can take several forms. For 
example, program personnel who solicit evaluation only to 
satisfy external funding agencies are engaging in "symbolic" 
use; the evaluation is only used to persuade the agency that 
the game is being played according to the rules. Lenihan 
(1977) provides an actual example of "symbolic" use. An 
evaluation demonstrated that several benefits could be realized 
by installing phones in a jail for inmate use. The 
evaluation's information was initially ignored. 

The [inmate] riot changed all that. When the time 
came, when overcrowding reached a breaking point, the 
knowledge produced from this research was put to use. 
It was not a sufficient cause for change, but in the 
end it did make a contribution (Lenihan, 1977, 
p. 583). 

The evaluation provided a face-saving justification for 
installing phones in the Jail, although the actual motives for 
doing so might have been less than scientific. Knorr (1977) 
indicates that roughly 10% of surveyed administrators report 
they have used evaluations to legitimize decisions. 

"Ritualistic" use is a misnomer, since the use can have 
deliberate and important program impacts. Presumably 
evaluation affects the behaviors of program personnel because 
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^^^y know program processes and impacts are being measured, 
^"""^^isingly , the results of this type of use have not yet been 
e°'Pi^ically investigated. 

Although these conceptualizations of use may accurately 
reflQ^^ Ureal world" evaluation dynamics, the shift from the 
1^°^^ traditional view of use does produce some problems. An 
expan(jg^ view of use makes it more difficult to study use 
phenomena. For example, "it is literally impossible 'to prove' 
^°o"^^ptual] use" (Fullan, 1980, p. . Nevertheless, the 
trade^Q^^ of measurability in return for a more realistic 
f^^'^^^ctive on use is probably worthwhile. As Braskamp and 
prown (igso, p. viii) have argued, "although the expanded 
fjefinj^^^^p makes utilization less dramatic and more difficult 
to expi 

icitly measure and demonstrate, it represents a view of 
evaluation in which the role of human interaction in the 
commutiicg^iQn process is .^iven more credence." 

MODELS OF "REAL WORLD^ DECISIONING 
'lust as the definition of evaluation use affects the 
^sses^jjjgj^^ of the levels of use, perceptions of how decisions 
^re m^(jg also affect judgments of the quality of use. As Wise 
(1978, 5) explains, "referring to administrators as 

'^®°^^ion-makers' and to what they do as 'decision-making' may 
ij3ve been a first step in creating the utilization problem, for 
yiS exjjQ^t tQ decisions being made by someone called a 

<^^^^^-^on-maker ." Thus many evaluators presume that evaluation 
(jUght used, because its evidence is rational. However, 
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as Weiner, Rubin, and Sachse (1977, p. ^) note, '^interestingly, 
their [evaluators' ] recommendations are based upon a faith in 
rationality, not upon evidence concerning the factors 
influencing the utilization of evaluative information." As 
Caplan (1980, p. 4) suggests, "obviously, utilization is not a 
simple process — bureaucratic, ethical, attitudinal, and social 
considerations take precedence over the value of information in 
its own right." 

Hayman (1979, p. 11) argues that the educational 
environment can be chaotic; consequently, administrators may 
only be able to invest careful and thorough thought in a few of 
their most critical decisions: 

[The "turbulent field" type of environment, one of 
four conceptualized environmental types,] appears to 
correspond to conditions facing education today. In 
a turbulent field, the accelerating -ate and 
complexity of interactive effects exceeds the 
component systems' capacities for prediction and, 
hence, control of the compounding consequences of 
their actions. 

Thus Simon (1957, p. 204) suggests that administrators may be 
forced to engage in "satisf icing , " i.e., the process "of 
finding a course of action that is *good enough.'" 
"Satisf icing" is tenable because administrators believe they 
can change most courses of action if decisions later prove to 
be seriously mistaken. 

Thus, evaluators must recognize that administrators at 
times may not behave "rationally," i.e., administrators' 
decisions may be rational only when they are viewed from within 
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the administrators' frames of reference. Of course, it is also 
true that some administrators are less rational than others. 
For better or worse, some "decision-makers pride themselves on 
'shooting from the hip' and would not have it any other way" 
(Cuba, 1969, p. 17). 

THE LITERATURE ON USE 

Overview 

A general consensus that there is a shortage of empirical 
use studies has emerged in the literature. For example, Davis 
and Salasin (1975, p. 626) reported that "a review of 1,200 
references on [knowledge] utilization contained only 2 1/2% 
which pertained to evaluation, again even in the broadest 
sense." Furthermore, 

While much has been said and written about the 
problems besetting evaluation and about the 
underutilization of evaluation information, very few 
empirical studies of evaluation utilization have been 
conducted. Most of the literature is anecdotal in 
form (Alkin & Daillak, 1979, p. 41). 

Shapiro (1979, p. 1) agrees: "The literature on utilization, 

both applied and theoretical, tends to be ad hoc and 

nonrigorous." Cook (1978, p. 14) has suggested that "the 

quality and imaginativeness of most (but not all) utilization 

studies leaves something to be desired." 

Of course, it is difficult to conduct good utilization 
research. For example, as Stevenson (1979, p. 3) notes, 
"verbal acceptance of findings may not be followed by 
appropriate action. Verbal rejection of findings may be 
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followed by actions which imply acceptance." Nevertheless, 
Caplan (1980, P- 9) hopefully overstates the case when he says 
that "the^e are no tested propositions or even a substantial 
consensus gs to what will work" in promoting use. 

Factor s AffectiM Use 

The literature ^^oes indicate that evaluation i^iformation 
is most likely to be used in either of two situations. 
Evaluator-s might wigj^ invest special effort in their work 
when these situations exist. First, £val^a_tive jjiformation is_ 
most lik^ Tj^ ^ used when a program ig novel and 

administra^ rs ££nnot rely extensively on their ejcper^ntial 
background^ in makj^ judgments- Matuszek and Holley (1977) 
report that, i" ^^^ir experience, 

The [evaluation] office has tended to get the most 
response to its evaluation information when it really 
does Jj^PJ^esent something the decision maker doesn't 
aXready I^i^qw. Thus, principals generally believe 
that they ^gve a fairly good feel for teacher 
attit^^es in their schools; a teacher attitude 
questionnaire is most likely either to confirm their 
feelings q^. to be rejected; student pattern of 
course choices (for example), on the other hand, is 
less likely to be at their fingertips. 

An empiricax study ty Granville (1977, p. 6) corroborates this 
conclusion : 

The t>®^t Way to express the magnitude of these 
effects ig to say that the Political and Social 
Influence factors swayed principals by about one and 
a half" intervals on a decision scale that ran across 
Six i'^J®'*^als. The Objective Evidence factor had a 
Separate effect of about one interval under the Novel 
Prograi? Condition. Under the Routine program 
Condition, gs I mentioned, objective evidence had 
Virtually no effect. 



ERIC 



Page 21 



Second, the literature suggests that evaluative 
information is most likely to be used when onl y moderate 
changes in the program are required and the environment is not 
extremely politicized . As Meltsner (1976, p. 9) indicates, 
there are wide variations across programs re irding the 
environments in which the programs operate: 

Sometimes the politics of a policy area are open and 
conflict ridden ; sometimes they are closed , 
involving technical issues and technical men. Some 
policy problems have a dimension of crisis to them. 
Others are chronic; they never seem to go away and 
are subject to incremental solutions. 

Consequently, Weiss (1972, p. 320) reports that "use of 

evaluation appears to be easiest when implementation implies 

only moderate alterations in procedure, staff deployment, or 

costs, Oi where few interests are threatened," 



LESS EFFECTIVE STRATEGIES FOR OPTIMIZING USE 



The literature includes several recommendations for 
optimizing use which it now appears may not be particularly 
helpful. For example, some evaluators consider evaluation 
models (cf. Stufflebeam, Foley, Gephart, Guba, Hammond, 
Merriman & Provus, 1971) panaceas which can be used to create 
us^:. As Brown ( 1980, p. 4) notes, 



For a time, it was hardly respectable to be an 
evaluator without having your own model. You at 
least had to be a disciple of a proponent of a new 
model that was on the "cutting edge" in order to 
maintain some semblance of self-esteem. It is 
interesting to observe that there were very few 
wounds inflicted by that "cutting edge." 

Due to these attachments to evaluation models, "the past few 
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years have witnessed the development and proliferation of 
myriad evaluation models" (Thompson, 1980, p. 59). Indeed, 
this proliferation leads to periodic efforts to synthesize the 
wide array of evaluation models (cf. Steele, 1973). 

Today, however, it is clear that evaluation models such as 
the CIPP model (Stufflebeam et al . , 1971), although important 
aids to earlier efforts at conceptualizing evaluation, are less 
important when viewed as guides for evaluator behavior. Thus, 
based on his case studies of evaluative practice, Alkin (1979, 
p- 7, emphasis in original) found that "none of the five cases 
involved the application of a formal evaluation model This 
finding was not an artifact of evaluator ignorance regarding 
the precepts of several evaluation models (Alkin, Daillak & 
White, 1979, p. 240). 

More recently it has been suggested that evaluative 
information would be more widely used if evaluators and 
administrators worked more closely together to formulate 
evaluation designs. For example, Suchman (1972, p. 67) has 
suggested that program goals must be explicitly specified prior 
to evaluation, because "unless a program can specify what value 
its activities are seeking to further, whether this be the 
amelioration of some specific social problem or the advancement 
of some broad humanistic goal, evaluation becomes meaningless." 
Similarly, Patton (1978, p. 202) has recommended collaborative 
development of evaluation designs; "it is crucial that 
identified decisionmakers and information users participate in 
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the making of measurement pnd methods decisions so that they 
understand the strengths and weaknesses of the data — and so 
that they believe in the data." Finally, Ross (1980, p. 66) has 
even suggested that evaluators and administrators should 
together specify decision rules in advance of program 
implementation; these rules specify what decisions will be 
taken if various evaluation results occur. 

Unfortunately, these strategies tend to look good on paper 
and work poorly in practice. For example, what Patton (1978, 
p. 100) has termed the "goals shuffle" can readily destroy 
efforts to estimate discrepancies between program goals and 
actual program outcomes: 

The goals clarification shuffle involves a sudden 
change in goals and priorities after the evaluator is 
firmly committed to a certain set of measuring 
instruments and to a research design. The 
choreography for this technique is quite simple. The 
top priority program goal is moved two spaces to 
either the right or left and four spaces backward. 

Rossi (1972, p. 229, emphasis in original) has argued that what 

might be called "methodology shuffles" can also occur if 

'pvaluation results prove to be unpopular: 

It is easy to attack the methodology of any study: 
methodological unsophisticates suddenly become 
experts in sampling , questionnaire construction , 
experimental design , and statistical analysis , or 
borrow experts for the occasion. 

Of course, decision rules tend to become obsolete once either 

goals or methodology shuffles have been performed. 
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In summary, the collaborative identification of goals, 
methods, and even decision rules is all well and good. 
However, it is important to recognize that these strategies are 
most likely to work m situations in which evaluation use is 
least threatened. More importantly, it must be recognized that 
these strategies do not themselves directly address the primary 
reasons why the situation exists in the first place. 

PLAUSIBLE STRATEGIES FOR OPTIMIZING USE 
The literature also includes some recommendations for 
optimizing use which, although thev are plausible and 
noteworthy, are easier said than done. For example, Havelock 
(1968) has suggested that use will be optimized if someone 
performs a »»linking agent" function. Hayman ( 1979, p. 1, 
emphasis removed) defines "linkage" as "a process of prcnoting 
knowledge utilization in educational organizations, and a 
^linking agent' is an individual or group which causes linkage 
to occur." It may be that evaluation is most likely to occur 
when an administrator who is unassociated with the evaluation, 
i.e., is perceived by fellow administrators as being objective, 
"adopts" an evaluation study and begins pushing for 
implementation of the study's results. The dilemma, of course, 
is that to be most credible this process must be spontaneous, 
so there are no guarantees that this form of linkage will 
occur. However, this form of linkage may be more likely to 
occur if the evaluator targets results toward a larger number 
of administrators. 
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It has also been suggested that utilization will be 

optimized if evaluative information is presented to 

administrators in a timely fashion. Randall (1969, p. 1) 
portrays what may be a common situation: 



There is a timeworn and oft-recurring spectacle of 
the frantic but finally productive 

researcher-evaluator , who rushed into the executive 
offices with his data analysis finally complete, his 
report prepared and in hand, only to find that the 
executives, several months previously, had made the 
important decisions that locked up the monies and 
committed the organization for:-., the ensuing months 
ahead . 

Unfortunately, it is not always possible to anticipate when 
information will be needed in service of decision making. In 
fact, as Brickell, Aslanian, and Spak (1974, p. 24) note, "he 
[the administrator] can never know when he will need it 
[evaluative information]. The process of government 
decision-making is not so orderly or regular that he can 
schedule his need for information." It is also i-^portant to 
recognize that timeliness is important for "int-t •• )..ficntal" use 
but may not be so for other types of use (Young & Comptois, 
1979). 



Finally, Johnston (1978, p. 1) has suggested that it is 
important to target evaluation toward identified 
administrators . 



There is a sort of ecology for each educational 
program, a network of people in different roles who 
influence (or are influenced by) the outcome of the 
program being evaluated. If this is true, and 
research utilization is the goal of the evaluator, 
then there are multiple audiences for an evaluation, 
not just the decision-maker who commissioned the 
evaluation. So the evaluator has a first task of 
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identifying who these other actors are. 
However, this effort can be frustrated by the complexity of the 
organizational network. As Randall (1969, p. 7) explains, 
"typically, the decision process in an organization involves a 
complex network of persons who have varying degrees of 
influence on the one who may have constituted authority to make 
any given decision." Tne situation is further complicated 
because, as Granville (1978, p. 29, emphasis in original) 
notes, an evaluation study "has to persuade not just the people 
who ostensibly make the decisions, but also the people they 
have to persuade." Thus, Alkin and Kosecoff (1973, p. 3) 
conclude that "identification of the program's decision 
maker(s) is perhaps the most elusive variable associated with a 
decision context." 

ESSENTIAL STRATEGIES FOR OPTIMIZING USE 
Holism is Crucial 

The literature also includes several recommendations for 
optimizing use which are generally essential to optimizing 
evaluation use. These recommendations involve identifying 
evaluation issues, acknowledging evaluation subjectivity, 
focusing on policy alternatives, and building rapport with 
administrators and program personnel. However, it is crucial 
that the evaluator use an holistic approach to adopting these 
strategies. As Weiner, Rubin, and Sachse (1977, p. 23) argu^i, 
"attempts to increase evaluative influence which focus on a few 
of these factors in isolation and which do not recognize the 
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highly complex and interactive system of forces °Or)strai"^"S 
evaluator activity are likely ^to fail to alter t^^ overall 
effects of the system." Patton (1978, pp. 19-20) conc^i^s, 
noting that "the overall problem of underutili^^tion 
evaluation research will not be solvecl by ^°°^Piling 
following some long list of evaluation proverbS ^^ioms." 

Issue Identification 

Evaluation results will enjoy ninstrumentQin and 
"conceptual" use only if the results address issues con^^®^" 
to administrators. As Alkin, Daillak, and White 0979, p, 238) 
note, «if the evaluation addresses a" p^ggsing °°noern ^ 
potential user, then the evaluation information i^ "or^ lively 
to draw, and hold, the user's attention." xhus ^^ttQ^ (l9'^8» 
P- ^3) goes so far as to suggest enhancing "utili^^^ j^qo by 
focusing on fulfilling one purpose extremely vJell» ^^gt at 
least the decisionmakers' [sic] central ^l^^^^ions 
answered." Alkin and Daillak (1979, p. 47) ^^^ue ^^^^ 
"evaluators who concentrate on the mandate^ eval^^ati^^j^ t^^^^ 
run the very real risks of losing the local audie"^®^-" 

However, it may be difficult to identify the i^sq^^ wtji^^ 
are most important to administrators. For exampl®' a uni^"® 
project designed to be particularly respons^^g to 
administrators' information needs, Fletcher (1972, P- 15) foUnd 
that "like teachers, administrators could ^ot often identify 
Kinds of data they could use. And in many cases the kij^^g they 
wanted „ere totally beyond our capacity t^? Provid®*" 
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Furthermore, the situation which Ingison (1979, p. 2) observed 
at the federal level may also occur in local education 
acrencies: "At the [National Science] Foundation (and elsewhere 
at the federal level, j suspect), the pressure is always on to 
get the evaluation study set up and brought in quickly 
(preferably yesterday).'' Despite these difficulties, to 
maxiiniee the liklihood that evaluative information will be 
used, it is recommended that 

Evaluators should concentrate evaluation efforts 
on the highest priority information needs of 
specific administrators, even if these needs 
require work beyond that mandated by external 
funding agencies. 

Since administrators are not always able to anticipate or 
articulate future information needs, evaluators "should 
anticipate questions and be proactive" (Law, 1980, p. 74). 
StaK^ (1973, p. 305) makes a similar point: "The evaluator, I 
thifi^' has a responsibility to snoop around and to guess at 
what decisions may be forthcoming* He should use these guesses 
to orient his evaluation plan." Gorham (1970, p. 104) argues 
that evaluators should "be clairvoyant about forthcoming 
issues. n Meltsner (1976, p. 127) cites the example of "one 
analyst [who] likes to follow his client around for a week and 
attend the meetings he does, and the like, to see what *is 
hurting the client.'" These arguments suggest that 
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2- Evaluators should identify some evaluation 
issues on the basis of emphatic and pro-active 
anticipation of administrators* future informa- 
tion needs. 

If these anticipated needs do not arise, evaluators need not 
highlight the results of the incuiries whic^i they initiated. 
In any case, evaluator credibility should be improved when 
administrators sense a sincere effort to be responsive, even 
though this anticipation will not always be precisely accurate. 

In addressing evaluation issues evaluators must also take 
into account the factors which determine the perceived salience 
of information. For example, as Brown (1973, P- 2) explains, 
"the question as to when and which levels of information school 
management needs depends on the length of time - a program has 
been in operation and the degree to which that project overlaps 
other programs within the school system." An administrator's 
location within the organizational hierarchy also affects 
extent of felt need for information (Braskamp, Brow*. & Newman, 
1978, p. 449). Furthermore, organizational position affects 
the type of information which is required; "those officials 
who are in a position to control the project from day to day or 
month to month and who are responsible for exercising such 
control have a far greater interest in monitoring reports than 
decision-makers at higher executive levels" (Brickell, Aslanian 
& Spak, 1974, p. 59)- Evaluators should also remember that 
"research is often most useful to those who do not have the 
authority to promote a policy, i.e., teachers" (Hamilton, 1980, 
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p. 7). In short, 

Tl Evaluators must tailor information studies to 
meet the different needs of various evaluation 
audiences . 

This may entail steps such as writing several versions of 
reports or conducting "extra" evaluation inquiries. 

Evaluation Subjectivity 

Many evaluators like to believe that evaluation is 
objective, apolitical, and appropriately empirical. All three 
of these beliefs can take the form of myths which hinder 
evaluation use. For example, Wilensky and LelDeaux ( 1958, 
p* 20) observe that "what the social scientist thinks of as 
'objective investigation' the practitioner often takes as 
'hostile attack.'" As Goodrich (1978, p. 632, emphasis in 
original) suggests, "what has happened is that we have tried to 
avoid the phenomenon of subjectivity in order to avoid the 
charge of subjectivity." However, Patton (1978, p. 237) argues 
that: 

The fundamental issue is whose values will bias the 
question , not whether or not questions will be 
biased. In a very real sense all questions are 
biased, but biased questions can be either open, or 
loaded . 

Meltsner (1976, p. 261) puts the matter nicely into 
perspective : 

Trust is also nourished by the analyst's attempting 
to be objective. This does not mean that either the 
analyst or the analysis can be objective in an 
absolute sense. Both do have their values. What it 
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does mean is that the analyst should attempt to give 
as straight an answer as he can. 

Some administrators perceive evaluation as a two-edged 
sword. The pretense of objectivity embues evaluation with 
credibility. The administrator who is confronted with an 
"unfavorable*^ result knows that any result can be attacked on 
several grounds. The administrator who is confronted with a 
"favorable" result may perceive the result as a weapon against 
adversaries. However, firm stances may not be taken until the 
results dictate them. To reduce some administrators' rather 
schizoid views of evaluation, and for the mutual benefit of all 
concerned , 

4. Evaluators should acknowledge the subjective T 
elements in their work, when, these elements are 
unavoidable, and offer their work merely as 
informed but not omnipotent support for 
decision-making . 

However, this recommendation must be implemented with care, 
because sometimes administrators misinterpret honest caveats as 
admissions of avoidable and serious evaluation flaws. 

Evaluation and Politics 

Although many evaluator5,, are not comfortable admitting it, 
evaluation cannot avoid being a political activity: (Weiner, 
Rubin & Sachse, 1977, p- 19). As Patton (1978, p. 46) 
explains : 

The traditional academic values of many social 
sc ientists lead them to want to be nonpoli tical in 
their research. Yet they always want to affect 
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government decisions. The evidence is that they 
cannot have it both ways. 

Isaac (1980, p. 3) concurs: 



Programs that are politically conceived and 
implemented, are also sustained and defended 
politically. It seems the better part of wisdom 
neither to be surprised nor offended by this 
phenomenon . 

According to Cohen (1972, p. 139), "to the extent that 
information is an instrument, basis, or excuse for changing 
power relationships within or among institutions, evaluation is 
a political activity." The implications of this situation have 
been identified by Dickey (1979, p. 3): 



Judgment has an awesome ring, anl it is not 
surprising that those who are being judged feel 
anxious, even threatened. Add to this the political 
context in which the process takes place (and there 
ia always a political context) and we have all the 
ingredients for dysfunctional communication — high 
levels of stress lea'ing to communication patterns 
arising from indivi-jal defense mechanisms. 



This does not mean that evaluators must themselves 
participate in political activity. However, as Meltsner (1976, 
D. ^3) Suggests, the effective evaluator "tries to understand 
political considerations and then to make them an integrated 
and explicit part of his analysis." According to Brown and 
Braskamp (198O, p. 93) "this means that the relationship 
between the evaluator and key program staff, and the 
^valuator's understanding of the organization in its internal 
and external poltical environment, are critical for successful 
Utilization." This suggests that 
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Evaluators should understand the politics of 
their agencies and attempt to meet the political 
needs of involved persons whenever doing so 
will not jeopardize the integrity of the 
evaluation . 



Evaluation and Empiricism 

Some administrators have a stereotypic view of evaluators 
in which evaluators are perceived as "technicians." Meltsner 
(1976, p. 23) summarizes the sterotype thusly: 

Unlike the messenger of ancient times, the technician 
does not fear for his head when he has to bring bad 
news. No, he would not soften his findings; he 
would do "the best analytical job that can be done in 
conformity with the principles of economics [or his 
discipl ine] ." He refers to his work as "honest 
analysis," and he complains about analysts having to 
provide justification for a decision that has already 
been made. 

However, the evaluation features which are persuasive to 
administrators stand in stark contrast to what the technician 
will emphasize. As Leviton and Hughes (1979, p. 23) suggest, 
"administrators prefer qualitative information to the 
quantitative data that evaluators frequently supply." 

Empirical research supports the view that administrators 
prefer qualitative information over quantitative information* 
Alkin's (1980b, p. 24) well-known naturalistic utilization 
studies yielded the conclusion that "little evidence was found 
in the case studies that research rigor was an important factor 
affecting utilization." Simulation research by Brown and Newman 
(in press) is even more dramatic regarding this point: 
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The simple addition of an inferential statement, such 
as "these differences were statistically significant 
at the .05 level" however, resulted in lower levels 
of agreement [with policy recommendations]. m fact, 
for three of four recommendations, tHe inclusion of 
the inferential statement resulted i^^ lev^l s of 
agreement lower than in the No Dat^ [Experimental] 
condition . 

However, it is important to note that the use of data does 
interact with n&ther result features in dete^^miJ^ing audience 
reaction (Brown, Newman & Rivers, 1980, p. 75), So a simple 
interpretation of these results is not possible- Of course, 
some administrator disdain for empiricism is rational if we 
acknowledge that quantitative forms of representation 
"inherently are insensitive to some of the significant aspects 
of classroom life" (Eisner, 1980, p. 11). 

This is not to suggest that evaluators should forego the 
quantitative aspects of their work; rathe^" "the central 
message in this regard is that it is not enough to conduct 
methodologically sound research" (Johnson, 1978> P* 12). 

5"^ Evaluators should emphasize both fornf^tiv^ 
process evaluation data and summative product 
evaluation data in their work. 

The two foci reinforce each other. The emphasis on process 
evaluation communicates both an understanding of program 
ecology and a commitment to program improvement; previous 
discussion has suggested that these messages ar^ essential- An 
emphasis on formative process data should then help make 
quantitative summacive results more credible* Of course, 
quantitative data must still be presented in concrete and 
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understandable terms, Evaluators might also do well to be sure 
that they understand the meaning and limits of their 
inferential methods (Carver, 1978). 

The Personal Factor 

The reader may perceive that some of the preceding 
prescriptions speak to the personality of the evaluator. The 
perception is accurate. The literature makes clear that the 
most critical determinent of evaluation utilization is what 
Patton (1978) has termed "the personal factor." As Cronbach et 
al. (1980, p. 6) summarize, "nothing makes a larger difference 
in the use of evaluations than the personal factor — the 
interest of officials in learning from the evaluation and the 
desire of the evaluator to get attention for what he knows." 
Evaluators sensitive to this factor will respond in two ways. 
First, following the suggestion of Patton (1978), evaluators 
will identify the evaluation's relevant decision makers and 
information users, and they will continually work on effecting 
utilization throughout the course of the evaluation. 

The second response suggested by the personal factor will 
aid this process. Several writers have noted the importance of 
good evaluator-client relations: 

The evidence on dissemination suggests that informal 
communication that cuts the red tape may enhance 
utilization , although quality of information may 
sometimes suffer and dissemination will be haphazard 
(Leviton & Hughes, 1979, p. 21). 

For while information is an essential resource for 
decision makers, the manner in which it is converted 
into policy is based as much or more on 
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interpersonal, organizational, and psychological 
factors than on the actual information itself 
(Guskin, 1980, p. 45). 

Utilization is usually the result of the relationship 
between the exaluator and the user more than anything 
else. If the user knows and respects the evaluator, 
utilization has its highest potential (Holley, 1979, 
p. 8). 

The major barriers to successful evaluation are not 
technical and methodological , though these are 
certainly important and worthy of further effort, but 
are rather the structural constraints and 
requirements and the interpersonal relationships 
which characterize the evaluation endeavor (Gurel, 
1975, pp. 27-28). 

To be effective. 



7"^ Evaluators must demonstrate to project personnel 
and administrators that they sincerely care about 
the needs of program staff and the program's 
clients . 



An example may illustrate how this admonition can work in 
practice. The first author was working in a local education 
agency as an evaluator of a court-ordered magnet school 
program* Since one of the program's functions was to promote 
racial integration of students, each magnet school was charged 
with actively recruiting students representing various ethnic 
backgrounds. After several months of operation it became clear 
that the staff of one project had deliberately avoided this 
responsibility in order to gain time to get their program on 
the ground. Although this decision was perfectly rational from 
the perspective of the program's staff, in light of the 
court-order this inactivity could have had serious implications 
for the school district. If "breathing time" was required, 
then the admissions phase of the program could be delayed, but 
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it was vital that recruiting activity begin as soon as possible 
so that lag times would be minimized. 

Consequently, it was necessary to meet with the program 
director and the administrator who supervised the director. 
The evaluator took this position: 

This is a potentially serious matter which I must 
present in my next process evaluation report. But 
how can I help you? Since your program has academic 
admissions criteria, would you like me to analyze the 
district's computer tapes from the last system-wide 
testing and generate mailing labels for eligible 
students in the grade levels you serve? You could 
write them and invite them and their parents over to 
see what your program has to offer. This way when I 
discuss your recruiting activities I can point to 
concrete evidence that the situation is being 
corrected . 

It was not technically the evaluator's job to generate mailing 
labels. However, actions like these are important because they 
communicate the evaluator's concern for the program and its 
clients . 

The "personal factor" in part means that evaluators try 
not to be threatening and authoritative; how evaluators 
comport themselves affects the psychological frameworks with 
which administrators interpret evaluative information. This 
conclusion may discomfort some evalntors who believe that they 
offer objective truth which they believe should have intrinsic 
value independent of evaluator personality or approach. 
Nevertheless, administrators have their own paradigms for 
viewing th . vnese paradigms are rational to them, and 

evaluators , 0 ^:0t that the manner in which the evaluator 
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interacts with administrators and staff will affect the 
credibility which the administrators vest in subsequent 
evaluation results. 

EVALUATION AND POLICY ANALYSIS 
Between ten and twenty years ago educational thinkers 
struggled to conceptualize evaluation and evaluation models. 
As Thompson (1980, p. 59) notes, "full consensus on these 
[conceptual] issues has certainly not emerged, but there does 
appear to be general agreement among theorists that evaluation 
is a process of providing information for decision-making, and 
that ultimately evaluation implies value judgments of worth." 
The conceptual distinction between "evaluation" and "policy 
analysis," however, has not yet been so completely resolved. 
There is some recognition in the literature that these two 
processes can be much the same thing, although some evaluation 
models imply "preoccupation with existing programs" and some 
policy analysis models "usually compare existing and 
hypothetical alternative program solutions" (Wholey, Scanlon, 
Duffy, Fukumoto & Vogt, 1970, pp. 23-24, emphasis in original). 

If evaluation is not a purely objective, purely empirical 
enterprise, then it is more reasonable to expect evaluators to 
consider policy alternatives in their work. Haller (1974, 
p. 403) observes that "evaluation problems concern decisions. 
Decisions presume the existence of alternatives, and so the 
purpose of evaluation is to help delineate alternatives and to 
orovide information to help decision makers arrive at more 
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rational choices." However, We^^* ^^^"^9, P* notes that many 
evaluators are not policy ori^^^ed, "evaluators not 
always— or even often— come ^^^h data that give explicit 

guidance for action." As ZeP®^^ ^^^80f ^* 1 ) "°^es, this 
situation is not entirely s^^^^^^^tiQ^y : •^Informing local 
decision makers that the local Ti^^^^I prog""^" is not effective 
does not give them the infort"^^^®"^ t^^gt they need to improve 
it." The consequences of this si^^^^^^^ ^^e explained by Deal 
and Rallis ( 1980, p. 210): isy ^^Scri^^^^S in a controlled 

form what already exists, sc:er.ti^^° ^^^e^f^^ «*oes not directly 
promote learning for the cra^^^'"^'^» It ^x^^t even help to 
maintain the status quo." 

Some empirical research indi^^^®^ ^^^^ evaiygtors do 
policy recommendations. por ^'^^'^jjie, Aikin, Kosecoff, 
Fitz-Gibbon, and Seligman (1974, P* ""^i ^^^rid that "thirty-nine 
of the 42 evaluators made "^^^^^^nd^^^^ns for project 
modification." But this resuJ^ ^^'^^badi^^^ other research 
(Goldberg, 1978, p. 16) which f^'^"^ ^^^t 

administrators do not 

perceive evaluation to be too helP^ui "finding Possible 

courses of action and choosi^^ aJt®''"9tive courses of 

action." This analysis suggests that, evaluation reports, 

"reasonable alternatives must b^ ^'"^^tcled' ^^ch alternatives 
have a good chance of being tri^^ '^^^n pr'^vided" (Ingison, 
1979, p. 4). In short, evaluati^" "^^^ht ^^^^ effective if 
it became more like policy anaiy^^^* 
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Isaac Assimov (1972) has written a short story which 
illustrates the psychology of this situation. A young 
scientist proves mathematically than an energy policy is going 
to result in the destruction of the universe. However, no one 
will believe his proof until decades later a mechanism for 
still generating huge quantities of energy without destroying 
the universe is also presented. The morale is that 
administrators may be more likely to act on evaluation 
information when they are presented with some policy 
alternatives to consider. 

Will administrators perceive recommendations as an 
unwarranted intrusion into the policy arena? They will if the 
evaluator's offerings take the form of grandiose schemes. They 
may not if specific policy alternatives are mentioned and the 
evaluator merely presents objective evidence, both pro and con, 
regarding the best predictions about likely program impacts. 
Some evaluators perceive administrators as being very jealous 
of their turf. In some cases these perceptions are fully 
justified. But many administrators do not feel threatened by 
good ideas and will take their wisdom where ever they can get 
it, especially if they believe that the recommendations are 
being offered in a sincere attempt to help as against being 
offered in a sincere attempt to be Machiavellian. However, as 
Newman and Brown^s (1980) results indicate, the utility of this 
effort will partly be determined by the situation-specific 
personalities and needs of thr involved administrators. 
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CRITIQUE OF PREVIOUS USE RESEARCH 
The previous research on the use of evaluative itif oj-ma^^®'^ 
has been dominated by two research approaches. There ^g^g t>een 
several reports of a limited number of cas© studied ^hj^^j^ pave 
not heen theoretically grounded (cf. Ali^i^, Dai^ak ^ Whi^®' 
1979). The other scenario of studies involves s^myiatio" 
investigations which at least purport to be Sro^j^^^gd 
communications or attribution theory (cf. Newm^n, ^ro^^i ^ 
araskamp, 1980, pp. 29-36). Simulation research typically 
presents administrators with a "simulated" evaluation ^epoJ.t i" 
which different report features, i-e., the sex the 
©valuator, are varied and the impacts of the variat^^^^jg are 
then assessed. Both research traditions have "nacj^ ma3°^ 
contributions to our understanding of the yge °^ Evaluative 
information. However, both scenarios have wea^^^es^^g whi^^ 
should be avoided in future use research. 

Alkin (1979, p. 13) has consistently argued that 

The forces which lead to utiligatio" .^''^ ind^®'* 
complex. This complexity in oombii^^tion ^^^^i 
current inadequate understanding evaj-Uat^^^ and 

utilization requires a method ologi^^^ ^roced^^^ 
sufficiently sensitive to capture nuan^®^ 
involved — naturalistic research j^g currently ^ m^st 

apporopriate tool for a study of ^Vaiuati*^" 
utilization . 

All-in, Daillak, and White (1979, p. 32, emphasis Original) 
are less restrained elsewhere when they arg^e that "ih^ choi5^ 

of ap propriate research strategies can be reduced to one clalil 

na_tur £iistic^ research methods ." However, this case stvj^jy ^o^^ 
can be criticized on at least three grounds. 
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f'irst, there is theory, or at least the beginnings of 
theory, which can be relied upon in conducting use research. 
Our untlerstanding of evaluation'- is reflected in fairly 
elal>oJ"ate conceptualizations of types of use, and in the 
conceptualizations of "types" of evaluators which Meltsner 
(1976) has offered. It is also fairly clear that 
coBiniunioation-related theories can greatly enhance our 
understanding of at least the reporting phase of the evaluation 
endeavor (Brown & Newman, 1979b). These theoretical frames are 
certainly not yet fully developed, but further progress in 
developing theory absolutely depends upon our testing and 
elaborating the constructs which we already have at our 
disposal. Thus, failure to theoretically ground naturalistic 
research seems unnecessary and unfortunate (Thompson, in 
press-b) . 

Second, most of the case study research has been based on 
post hoo interviews with evaluators and evaluation clients. 
Leviton and Hughes (1979, P- 15) have commented on the dangers 
of restrospective research methods: 

Given officials' faulty memories, retrospective 
research may be biased in favor of a few, dramatic 
instances of use, rather than frequent but modest 
ones. 

Retrospective methods are economical, but their limitations 
must be acknowledged. 
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Third, some case study research can be criticized for the 
way it has been reported* Although some authors (Meltsner, 
1976; Patton, 1978) have integrated themes and case study 
evidence into a persuasive and incisive analysis, some 
researchers have presented case study evidence in 
non-integrated blocks of detail absent any themes. Thus some 
naturalistic research represents a ^jor return on investment. 
Ironically, if an evaluator communicated evaluative data as 
poorly as some case study research has been communicated the 
evaluative information would never be used. 

The simulation research can also be criticized on three 
grounds. ^irst, although this research is typically 
represented as being theoretically grounded in communications 
or attribution theory, this grounding has too frequently taken 
the form of "name dropping" the theory^ s title without invoking 
specific propositions of the cited theories. This is 
unfortunate since these theories do incorporate reasonably 
specific propositions about phenomena (see Davis & Salasin, 
1975, p. 641). An example proposition is offered by Thompson 
(1971, p. 185) who argues that communicators should "provide 
rationalizations for listeners who are unwilling to admit that 
socially disapproved motives are responsible for their beliefs 
or actions." 

A second criticism of the simulation research is that some 
investigators have used "samples of convenience" in their work. 
It is unlikely that the business majors or education students 
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who happen to enroll in graduate courses are representative of 
the administrators to whom the researchers sometimes attempt to 
generalize. This criticism is independent of and probably more 
telling than the recognition of some researchers (cf. Brown & 
Newman, 1979a, pp. 6-7) that simulation research may not 
perfectly generalize to natural ecologies. 

Third, simulation research may be criticized on the 
grounds that it typically focuses on the report phase of 
evaluation and we know that "what happens before the final 
report is written will usually determine utilization" (Patton, 
1978, p. 266). Still, it must be acknowledged that if we 
really want to optimize evaluation use then it may be necessary 
to emphasize all phases of the evaluation endeavor, including 
those which ar3 relatively less important in determining use. 

A RESEARCH A.GE-NDA FOR THE FUTURE 
Despite dramatic progress in achieving understanding of 
use phenomena, much remains to be learned. Several priorities 
for a research agenda can be identified: 

1. Prospective case studies of use should be a high priority 
for future research. Connor (1979, p. 16) makes this 
point quite well: 

The absence of studies with a "current" time 
orientation is a serious missing link in the 
utilization research chain... This orientation is 
essential if we are to obtain the most accurate 
information about utilization. Retrospective 
studies, while useful, are subject to biases directly 
related to the type of use which has occurred. 

2. Researchers need to determine whether or not school 
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personnel can effectively be trained to make more optimal 
use of evaluative information. For example, research is 
needed to determine if it is feasible to help 
administrators increase "problem solving capabilities and 
ability to express and articulate [information] needs" 
(Haenn, 1980, p. 13) - Training of LEA evaluation staff 
members should also be explored, since "there is a 
striking absence of any formal training in evaluation for 
staff" (Lyon, Doscher, McGranahan & Williams, 1978, 
p. 70). 

For example, will training in the communications and 
rhetorical traditions involving such matters as a fortiori 
logic help evaluators be more effective? Quade (1964, 
p. 173) explains this logic: 

To make an analysis a fortiori, we bend over backward 
in making the comparisons to "hurt" the system we 
think is best and to "help" the alternative s>^tem. 
If it then turns out that after we have done this we 
can still say we prefer the handicapped system, we 
are in a strengthened position to make [policy! 
recommendations . 

3« Further insight into how evaluators perceive themselves 

and are perceived by administrators is needed, if we 

accept that the personal factor is a critical determinant 

of use. Some research of this sort has been conducted 

(cf. Thompson, in press-a) , but we need a broader 

understanding of these perceptions similar to that which 

has been achieved in disciplines such as teacher education 

(Miller, Thompson & Frankiewicz, 1975). 
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4. More use research which focuses on principals and teachers 
as users is needed* Several researchers have involved 
persons in these roles as subjects (cf. Thompson, 1981), 
but the use of persons in these roles as subjects is not 
in proportion to the influence which these individuals 
exert over program operation. For example, regarding 
principals, Lipham (1980, p. 83; notes that "the 
leadership behavior of the principal is a powerful factor 
which influences the adoption and institutionalization of 
3n educational change." 
5* radical as it may sound, research on the nierits of 
mandated evaluation is also needed • Do programs which are 
not evaluated differ from comparable evaluated programs 
regarding either program processes or productivity? At 
some point the notio-n of mandated evaluation itself needs 
to be evaluated. 

MAKING UTILIZATION HAPPEN 
Persons who discuss evaluation use tend to lay the blame 
for non-use at someone else's doorstep. However, 
administrators must assume some responsibility for making s^fe 
that evaluative information is usable and used (Meltsner, 
1976)* So too evaluators must accept some responsibility for 
making use happen. As Polivka and Steg (1978, p. 697) argue: 

Traditionally, the evaluator has been very hesistant 
to claim any responsibility for the use of his/her 
findings. This approach has helped make it very easy 
to ignore evaluation results. 
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Both administrators and evaluators need to recognize that 
the responsibility for use is not a "zero-sum game^ in which 
responsibility can be divided up and will always total to 100%. 
We would be better off if both administrators and evaluators 
assumed 60 or 80% of the responsibility for optimizing use. 
Even then, as Patton (1978, p. 96) notes, "increasing 
- utilization potential does not guarantee utilization of 
findings. There are no guarantees." 
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